Why Big Data Needs “Agile” Help

Analytics professionals have the curiosity of a cat, the cool logic of Spock and the problem-solving skills of MacGyver. They are the rock stars of the machine-learning era. But put them in a room together and ask them to deliver a product of business value, and they often look anything but Masters of the Universe.

To be fair, software development projects are rife with uncertainty. Analytics projects are particularly challenging because the raw material—data—can be hard to get and turn into something usable. That’s why an adaptive project management approach is crucial, and why “agile” is the discipline perfectly suited to analytics.

Agile is a project management method that involves adaptive planning and continual improvement. It demands a massive dose of cross-functional collaboration and flexibility. It began as an iterative approach to managing software projects that relied on short development cycles and incorporated stakeholder feedback with every iteration. Agile has since been adapted for a host of other management processes.

“I think agile is probably the way to go for many types of analytics projects,” says Sandy Staples, a professor at Smith School of Business. “If you're modelling or working with data, it’s really a journey of discovery. You need some way to keep on track and to talk to your business partners to determine where to go next.”

Staples and Smith PhD student Mikhail Tsoy took a deep dive into four analytics projects that used aspects of agile in order to find what made them successful. In this conversation with Smith Business Insight, Staples discusses what they learned.

In what ways are analytics projects different from other types of projects?

I think it’s the degree of uncertainty. Other types of projects have plenty of uncertainty but analytics generally has uncertainty arising from two dimensions. One is, How do we answer the business problem and can we actually get any advantage from the data? There’s also uncertainty in terms of how to answer that question because there are various options in terms of modelling. A related uncertainty is around the quality of the data. Can we get the data we need? Is it clean and how do we make meaning from it? The degree of uncertainty is quite high in these projects.

There are at least a dozen project management methodologies for software development projects. What makes agile particularly effective for analytics? 

It doesn’t matter whether you’re running scrum or other agile methodologies, the major benefit is creating a strong partnership with your business. They have to be there to help you, to decide whether what you’re doing is adding any value. Agile allows for frequent reviews and feedback mechanisms: You pick a chunk of work that seems fairly certain to deliver some value, finish it, show it to stakeholders, and then decide what should be next on the list. You keep going through the iterations. These iterations are like cycles of discovery with very distinct points where you stop and consider what you’ve learned.

Then there are specific roles in agile that create linkages with business partners. In scrum language, they call it the product owner, someone from the business who is responsible for the list of requirements. That’s a very flexible list, and they keep it up to date and prioritize it so that when the team is ready to do the next batch of work, they’re ready. 

It’s the nature of analytics where sometimes you have to let the data show the possibilities. The data scientists have to decide if it makes sense, but you really need the business knowledge there too. It’s rare to find a data scientist who also has business domain knowledge.

You looked at four analytics projects that used aspects of agile, not always effectively. What success factors seemed to jump out?

The main one was around strong customer interaction. In the most successful project we looked at, the analytics group would not do the project unless the customer side was willing to dedicate two or three people to be on the team full time. They were making sure that whatever model was produced addressed business needs, and that throughout the project the business side would learn what was coming. Part of the mandate of the analytics group was to help other business units understand what could be done with data and how to do it in an agile, flexible way. 

The weakest project we looked at was launched with good intentions. But it became hard for the project team to get the attention of the product owners and senior leadership. One of their business reps told us that he didn’t see the point of going to the daily meetings and basically dropped out.

The really successful project [group] also did an incredible job of selecting projects they wanted to take on. Part of that was stakeholder commitment, but they also did feasibility analyses in terms of whether they had data that would address the problem. On one of the more unsuccessful projects, managers were surprised by all the data challenges. They didn't do much upfront work in terms of understanding whether they could get what they needed. This contributed to delays with the project.

Would it be fair to say that analytics managers new to agile should buy into the full methodology rather than implement it ad hoc?

I totally agree with that, especially if you haven’t done it before. Start with what others have learned from many years of running projects that require learning and flexibility about some approaches that we know will work. Then over time you can customize the approach if a part isn’t needed or adding any value.

There’s a different set of values that you have to buy into for agile. It’s accepting that you will learn as you go along. For companies that are used to a predictive approach, where a project is well defined upfront, with detailed plans and the team expected to go away and do it, it’s a very different mindset and level of involvement.

Did you come away with any insights that caused you to question your assumptions?

It's easy to be negative about the least successful project, in terms of implementation. To be fair, they took on the riskiest project, but it was the nature of that department. They were trying something very innovative. They believe they learned a fair bit but the organization didn’t set up the agile mechanisms to help the team work through uncertainties. 

Now, the most successful project followed a structured approach and they were very careful what they picked. Does that actually hinder their ability to innovate? Did they discard some risky ideas that might have paid off for the company? They wouldn’t know because they didn’t try.  

The puzzle for me is, how does a company balance this ability to innovate and take risks versus controlling and guaranteeing a little more what they will get out of their investment?

This should be something that is part of the discussion with your business partners. It helps if you can have shared perceptions of risk and an understanding of what everyone will get out of the project at the end. Also, to take whatever you’ve done and go in a slightly different direction when it’s not working. We saw that with one of the projects. Once they learned that their initial idea was not going to work, they didn’t abandon it. They pivoted and got some value out of their investment.